Serial No. 09/546,392 

REMARKS 

INTRODUCTION 

Claims 1-22 were previously pending and stand rejected. 
Claims 1, 2, 6, 7, 11-13, 17, 18 and 22 have been amended. 
Claims 23-26 have been added. 

Claims 1-26 are therefore now pending and under consideration. 

No new matter is being presented, and approval and entry are respectfully requested. 

PRIOR ART: SUZUKI 

Suzuki discusses a system for imaging a sample object to determine its construction, 
which is the arrangement of its parts. Images of the sample object are captured from different 
directions. Image data such as edges and heights are extracted from the images and then 
compared to pre-stored three-dimensional part-form data to determine the construction of the 
sample object, the construction being the three-dimensional orientation and position of the parts 
as arranged to form the sample object. The construction of the sample object is then used to 
derive an instruction plan for a robot. 

For the purpose of discussion, it is noted that the rejection compares, in the present 

claims: 

"reference models" to Suzuki's part-form data (3d data of parts) stored in advance; 
"reference object" to Suzuki's sample element 10, which is constructed of parts; and 
"object of detection" to Suzuki's sample element 10. 

FORMALITIES CORRECTED 

Various formalities in the claims and specification have been corrected. Withdrawal of 
the object to the claims is respectfully requested. 
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REJECTIONS UNDER 35 USC § 112, SECOND PARAGRAPH 

In the Office Action, at page 3, claims 8-10 and 19-21 were rejected under 35 U.S.C. 
§112, second paragraph, for the reasons set forth therein. The rejection states that in claims 8- 
10 and 19-21 , "said second image capturing device" does not have antecedent support. 

Claims 8-10 depend from claim 6 or claim 7. Claim 6 and claim 7 both previously recited 
the system of claim 1 "further comprises a second image capturing device". Claims 6 and 7 
have been amended only to improve their form and without changing their scope. Similar 
remarks apply to claims 19-21 and their parent claims 17 or 18. 

Withdrawal of the rejection is respectfully requested. 

REJECTIONS UNDER 35 USC §§ 102 AND 103 

In the Office Action, at pages 3-5, claims 1-4 and 12-15 were rejected under 35 U.S.C. 
§ 102 as anticipated by Suzuki. Claims 6, 7, 17 and 18 were rejected obvious over Suzuki in 
view of Maeno. Claims 8-10 and 19-21 were rejected as obvious over Suzuki in view of 
Sakakibara. Claims 1 1 and 22 were rejected as obvious over Suzuki in view of Maeno and 
Kelley. These rejections are traversed and reconsideration is requested. 

SUZUKI'S REFERENCE MODELS (PART-FORMS) ARE NOT "BASED ON IMAGE DATA OF A 
REFERENCE OBJECT", AND DO NOT HAVE ASSOCIATED CAPTURE DIRECTIONS. 

Claim 1 recites "storing reference models each created based on an image of a 
reference object". The rejection compares the reference models to Suzuki's part-form memory 
19. However, the part-form data in Suzuki's part-form memory 19 is not based on captured 
image data. In contrast, Suzuki describes such part-form data as "prestored" "3-dimensional 
forms of the parts" that make up the sample 10. Suzuki also describes the part data as "form 
data of the parts included in the sample [and] prestored in part-form memory 19" (col. 6, lines 36 
and 37). In Suzuki, the three-dimensional part-forms are compared to (not "based on") captured 
images by using height data derived from the captured images. Suzuki does not discuss what 
its part-form data is based on. Such three-dimensional part-form data, being of known parts, is 
most likely CAD or 3d model data in existence before any image capturing data. The part-forms 
are not based on images of the same. 

Furthermore, claim 1 recites for each reference model storing "information of the 
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capturing direction of its associated image". In Suzuki, the image data obtained "from various 
directions" is used only to detect "3-dimensional positions of vertex edges". Nowhere does 
Suzuki discuss or suggest storing and associating the various capture directions with the pre- 
determined part-forms (reference models). Suzuki does not suggest any associating of the 
capture directions. 

The distinction above can be further understood by considering that in claim 1 for 
example there is generally one object or one shape of object in use. The object imaged from 
different positions is the "reference object", and the captured images of the reference model are 
used to create the reference models. The object whose position/orientation is to be determined 
is the "object of detection". The "object of detection" is either the same object as the reference 
object, or is the same shape thereof ("reference object being the object of detection or having a 
shape substantially identical to that of the object of detection"). 

As noted above, the rejection compares: claim 1's "reference models" to Suzuki's part- 
form data (data of parts); and claim 1's "reference/detection object" to Suzuki's sample element 
10, which is "constituted by a plurality of [the] parts having known forms" (Abstract). However, in 
Suzuki, it is the "sample product [10] constituted by [the] plurality of parts" that "is measured by 
imaging the sample product from a plurality of directions" (Abstract). Therefore, the part-forms 
cannot correspond to the reference models because they are not created from images of the 
reference object horn different directions. 

Similarly, claim 12 recites capturing information of the orientation of the robot when the 
image of the reference object is captured. This is then used as one of the bases for later 
determining the orientation/position of the robot operation to be performed on the object of 
detection. Suzuki does not discuss so associating and then using the orientation of an image 
capture device. 

SUZUKI DOES NOT DETERMINE AN OPERATION ORIENTATION/POSITION BASED ON 
CAPTURED DIRECTION AND ORIENTATION 

One aspect of the presently claimed invention is that it selects a matching reference 
model having an image of the object and an associated orientation of the object. Performing an 
operation on an object generally benefits from knowing the orientation of the object relative to 
the robot. Thus, by matching an image of a reference model, the reference model's associated 
orientation can be used to determine the orientation of the object to be detected or operated on. 
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In Suzuki, the discussion is primarily about how to plan the construction or assembly of 
the sample object using the parts thereof by first imaging the assembled sample object and then 
computationally deconstructing the sample object. There is no discussion of how to "determine 
orientation, or orientation and position of an operation to be performed by the robot ... based on 
said one reference model and ... its associated capturing direction, and based on information of 
the orientation of the robot operation with respect to the object of detection matched with said 
one reference model". 

More specifically, in Suzuki, three dimensional positions of the vertices and edges of the 
sample product (10) (constituted of a plurality of parts), which are determined based on analysis 
of the two dimensional image data imaged from various directions by a pair of CCD cameras 
according to the abstract and col. 4, lines 5-9. Thus, Suzuki fails to discuss or suggest 
performing matching processing on an image of the object of detection with reference to models 
predetermined (captured and stored) based on image data of a reference object captured in a 
plurality of different directions so as to determine orientation/position of the object of detection, 
as recited in claims 1 and 12. 

Withdrawal of the rejection of claims 1 and 12 is respectfully requested. 

ANY FURTHER REJECTION REQUESTED TO BE MADE NON-FINAL 

Applicant notes that the amendments to the claims are intended to improve their 
readability. No new features have been added and the scope of the claims has not been 
significantly narrowed. Any new grounds for rejection of the claims cannot have been 
necessitated by their amendment herein. Applicant respectfully requests any future Office 
Action with new grounds for rejection be made non-Final. 

DEPENDENT CLAIMS 

The dependent claims are deemed patentable due at least to their dependence from 
allowable independent claims. These claims are also patentable due to their recitation of 
independently distinguishing features. For example, claim 2 recites "said reference models are 
obtained from a part of the image data of the reference object". This feature is not taught or 
suggested by the prior art. Withdrawal of the rejection of the dependent claims is respectfully 
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requested. 



NEW CLAIMS 

New claim 23 has been added to recite another aspect of the present invention in which 
a particular method of acquiring the reference images/information is not required. Claim 23 is 
distinguishable because it uniquely recites "determining a known arrangement of the robot 
relative to the unknown arrangement of the workpiece based on information indicating the 
known arrangement of the robot, and based on the reference arrangement information 
corresponding to the found reference image". Claim 23 is supported at least by Figures 5 and 6 
and element 208. 

CONCLUSION 

There being no further outstanding objections or rejections, it is submitted that the 
application is in condition for allowance. An early action to that effect is courteously solicited. 

Finally, if there are any formal matters remaining after this response, the Examiner is 
requested to telephone the undersigned to attend to these matters. 

If there are any additional fees associated with filing of this Amendment, please charge 
the same to our Deposit Account No. 19-3935. 



Respectfully submitted, 



STAAS & HALSEY LLP 




700 Eleventh Street, NW, Suite 500 
Washington, D.C. 20001 
(202)434-1500 
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VERSION WITH MARKINGS TO SHOW CHANGES MADE 

IN THE SPECIFICATION: 

Please AMEND the paragraph that begins at page 9, line 13, as follows: 

Scanning range (measuring range) on an object is set in advance, and an inclination 
angle 0x t 9y of the mirrors is controlled discretely. As shown in FIG. [7] 8, the scanning is 
performed from a point (1,1) to a point (1, n), from a point (2,1) to a point (2, n), .... from a point 
(m, 1) to a point (m, n) on the X-Y plane within the scanning range, to measure three- 
dimensional positions of each reflected point on the object. Also, a distance Z (i, j) between the 
sensor and the reflection point (i, j) on the object is obtained and stored in the [RAM38] RAM39 
of the image processing apparatus 30. Thus, the image data is obtained as two-dimensional 
arrangement data including the distance data Z (i, j) between the sensor and the reflection point 
on the object, as shown in FIG. 9 

Please AMEND the paragraph that begins at page 19, line 20, as follows: 

If it is determined in Step 204 that an object having a matching value equal or greater 
than the set value with respect to any of the reference models is detected, the procedure 
proceeds to Step 207 to perform matching processing on the two-dimensional data of the 
detected workpieces W, using every taught mode. In Step 208, the reference model having the 
most large matching value in the pattern matching processing is selected, and the relative 
position/posture of the workpiece W with respect to the camera 21 is determined based on the 
first-sensor relative position/posture, i.e., the relative position/posture of the camera and the 
reference workpiece stored for the selected reference model, and position, rotation angle and 
scale of the image of the workpiece in the matching processing. Also, data of the workpiece- 
robot (second sensor) relative position/posture associated with the selected reference model, 
which represent the position/posture of the second sensor 22 to be situated with respect to the 
workpiece are read from the nonvolatile memory [37] 38 (Step 208). 

Please AMEND the paragraph that begins at page 21 , line 1 , as follows: 

The position/posture of the second visual sensor 22 to be situated for a subsequent 
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operation in the world coordinate system is determined based on the determined 
position/posture of the detected workpiece W in the world coordinate system and the workpiece- 
robot (second sensor) relative position/posture data (approach vector) (Step 210). The 
processor [1] 31 operates the robot to situate the second visual sensor 22 to have the 
determined position/posture, and outputs a measuring command to the second visual sensor 22 
(Step 211). 

Please AMEND the paragraph that begins at page 21, line 26, as follows: 

In the case where a stack of the workpieces can not fall within the field of view of the 
camera [20] 21/22 , or in the case where it is not necessary to capture an image of a workpiece 
behind other workpieces by changing the orientation of the camera, the procedure may return to 
Step 200 when it is determined "Yes" in Step 205, to move the camera to another 
position/posture at which an image of the objective workpiece can be captured. 



IN THE CLAIMS: 

Please AMEND the claims in accordance with the following: 

1 . (ONCE AMENDED) A robot system having an image processing function for 
determining [posture] orientation , or [posture] orientation and position of a robot operation on an 
object of detection, the system comprising: 

a robot; 

a first image capturing device; 

a memory storing reference models , each created based on an image [data] of a 
reference object captured by said image capturing device in a [plurality of] different [directions] 
direction , and for each reference model storing information of the capturing direction of its 
[directions to be respectively] associated image [with said reference models,] and information of 
an orientation of the robot [operation] with respect to the reference object, said reference object 
being the object of detection or [an object] having a shape substantially identical to that of the 
object of detection; and 

a processor to perform matching processing on [image data containing] an image of the 
object of detection (captured by said firet image capturing device) [with] using said reference 
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models to select one of the reference models whose [an] image of [an] the reference object 
matched with the image of the object of detection [one of said reference models], and to 
determine the orientation, or the orientation and position of [an] the robot operation to be 
performed [by the robot] on the object of detection, the determining based on the selected image 
of the reference object, based on said one reference model and the information of [the] its 
associated capturing direction^ and based on the information of the orientation of the robot 
[operation] with respect to the reference object that is associated with said one reference model. 

2. (ONCE AMENDED) A robot system having an image processing function 
according to claim 1 , wherein said reference models are obtained [form] from a part of the image 
data of the reference object. 

3. (UNAMENDED) A robot system having an image processing function according 
to claim 1 , wherein said reference models are obtained by processing the image data of the 
reference object. 

4. (UNAMENDED) A robot system having an image processing function according 
to claim 1, wherein said first image capturing device comprises a camera for capturing two- 
dimensional image data. 

5. (UNAMENDED) A robot system having an image processing function according 
to claim 4, wherein said image data of the reference object are captured by said camera from a 
predetermined distance. 

6. (ONCE AMENDED) A robot system having an image processing function 
according to claim 1, [wherein said robot system] further comprising [comprises] 

a second image capturing device ; wherein [and] 

said robot situates said second image data capturing device to have said determined 
orientation or to have said determined orientation and said determined position with respect to 
the object, and wherein 

said processor processes second image data captured by said second image capturing 
device to detect position and/or posture of the object with respect to said second image data 
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capturing device. 

7. (ONCE AMENDED) A robot system having an image processing function 
according to claim 1 [, wherein said robot system] further comprising [comprises! 

a second image capturing device for obtaining three-dimensional position; [and] wherein 
said robot situates said second image data capturing device to have said determined 

orientation or to have said determined orientation and said determined position with respect to 

the object, so that said second image data capturing device is directed to a characterizing 

portion of the object; and wherein 

said processor detects three-dimensional position and/or posture of the object based on 

three-dimensional position of said characterizing portion obtained by said second image 

capturing device. 

8. (ONCE AMENDED) A robot system having an image processing function 
according to claim 6 or 7, wherein said first image data capturing device is used as said second 
image data capturing device. 

9. (UNAMENDED) A robot system having an image processing function according 
to claim 6 or 7, wherein said second image capturing device comprises a three-dimensional 
visual sensor of spot-light scanning type capable of measuring distance between the sensor and 
an object. 

10. (UNAMENDED) A robot system having an image processing function according 
to claim 6 or 7, wherein said second image data capturing device comprises a structured-light 
unit for irradiating a structured light on an object and capturing an image of the object including 
the irradiated light on the object. 

11. (ONCE AMENDED) A robot system having an image processing function 
according to claim 7, wherein said robot operation is an operation of picking up at least one 
object [form] from a plurality of objects overlapped with each other. 

12. (ONCE AMENDED) A robot system having an image processing function for 
determining [posture] orientation , or [posture] orientation and position of a robot operation on an 
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object of detection, the system comprising: 
a robot; 

a first image capturing device; 

a memory storing reference models , each created based on images [image data] of each 
of different kinds of reference objects captured by said first image capturing device, and storing 
[information] indicia of the kinds respectively associated with said reference models, and 
information of a different orientation of the robot [operation] with respect to each of the different 
images of the reference object of each kind, each of said kind of reference objects potentially 
being the object of [operation] detection [of each kind] or an object having a shape identical to 
that of the object of detection [operation of each kind]; and 

a processor to perform matching processing on [image data containing] an image of the 
object of detection [operation] (captured by said first image capturing device] [with] using said 
reference models to select an image of [an object matched with] one of said kinds of reference 
models, and to determine the orientation, or the orientation and position of the robot operation to 
be performed on the object of detection, the determining based on the selected image of the 
reference object, based on said one reference model, based on the [information] indicia of the 
kind associated with said one reference model and the information of the orientation of the robot 
[operation] with respect to the reference object associated with said one reference model of said 
one kind. 



13. (ONCE AMENDED) A robot system having an image processing function 
according to claim 12, wherein said reference models are obtained [form] from a part of the 
image data of the reference object. 

14. (UNAMENDED) A robot system having an image processing function according 
to claim 12, wherein said reference models are obtained by processing the image data of the 
reference object. 

1 5. (UNAMENDED) A robot system having an image processing function according 
to claim 12, wherein said first image capturing device comprises a camera for capturing two- 
dimensional image data. 

16. (UNAMENDED) A robot system having an image processing function according 
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to claim 15, wherein said image data of the reference object are captured by said camera from a 
predetermined distance. 

17. (ONCE AMENDED) A robot system having an image processing function 
according to claim 12, [wherein said robot system] further comprising [comprises! a second 
image capturing device, wherein 

said robot situates said second image data capturing device to have said determined 
orientation or to have said determined orientation and said determined position with respect to 
the object, and wherein 

said processor processes second image data captured by said second image capturing 
device to detect position and/or posture of the object with respect to said second image data 
capturing device. 

18. (ONCE AMENDED) A robot system having an image processing function 
according to claim 12, [wherein said robot system] further comprising [comprises] 

a second image capturing device for obtaining three-dimensional position; [and] wherein 
said robot situates said second image data capturing device to have said determined 

orientation or to have said determined orientation and said determined position with respect to 

the object, so that said second image data capturing device is directed to a characterizing 

portion of the object; and wherein 

said processor detects three-dimensional position and/or posture of the object based on 

three-dimensional position of said characterizing portion obtained by said second image 

capturing device. 

19. (UNAMENDED) A robot system having an image processing function according 
to claim 17 or 18, wherein said first image data capturing device is used as said second image 
data capturing device. 

20. (UNAMENDED) A robot system having an image processing function according 
to claim 17 or 18, wherein said second image capturing device comprises a three-dimensional 
visual sensor of spot-light scanning type capable of measuring distance between the sensor and 
an object. 
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21 . (UNAMENDED) A robot system having an image processing function according 
to claim 17 or 18, wherein said second image data capturing device comprises a structured-light 
unit for irradiating a structured light on an object and capturing an image of the object including 
the irradiated light on the object. 

22. (ONCE AMENDED) A robot system having an image processing function 
according to claim 18, wherein said robot operation is an operation of picking up at least one 
object [form] from a plurality of objects overlapped with each other. 

23. (NEW) A method for automatically determining an arrangement of a workpiece 
relative to a robot, the robot comprising an imaging device affixed to an arm, the method 
comprising: 

storing reference images of the workpiece or an object so shaped (workpiece/object) and 
reference arrangement information indicating arrangements of the robot and workpiece/object 
relative to each other when the images were captured; 

from a known arrangement of the robot, capturing a working image of the workpiece with 
the imaging device; 

finding one of the reference images that has a closest match to the working image; and 
determining a known arrangement of the robot relative to the workpiece based on 

information indicating the known arrangement of the robot, and based on the reference 

arrangement information corresponding to the found reference image. 

24. (NEW) A method according to claim 23, wherein reference images and reference 
arrangement information is obtained for workpieces/objects of different shapes, and wherein the 
finding comprises first determining that a reference image of one of the different shapes 
matches the working image of the workpiece, and then finding one reference image of the shape 
that best matches the working image. 

25. (NEW) A method according to claim 23, wherein the robot is used to capture the 
reference images, and wherein the reference arrangement information represents arrangements 
of the robot when capturing the reference images. 

26. (NEW) A method according to claim 23, wherein a second imaging device is 
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affixed to the arm and is used to determine additional arrangement information used to 
determine the known arrangement of the robot relative to the workpiece. 
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